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HEAR/HERE TO SEE: 
ALGORITHMS OF WESTMINSTER 


is a journey through the political heart 

of London that reveals the new forms of 
algorithmic governance and decision-making at 
work behind historical facades of power. 


Guided by the conversations of three teenagers, 
Eliza, Ruby and lo, the walk begins in sight 

of iconic seats of legislative, judicial and 
ecclesiastical authority. 


Along the route, the teenagers encounter 
increasingly prevalent, but far less apparent, 
modes of management and control made 
possible by ubiquitous networked computing. 


Their exploration of these new mechanisms and 
associated issues begins with an account of 
what exactly an algorithm is, informed by the 
methods of church bell-ringing. 


SCAN TO DOWNLOAD THE APP 


PARLIAMENT SQUARE 


BOVE LL ules lady lees 


. what comes next? 


R, 4S Big Ben! 
E Someone’s ringing those bells every 15 
minutes? Are they in the tower all day? 


R No, Big Ben is played by a machine. 
But the bells in Westminster Abbey are 
rung by hand — ten different bells, ten 
bellringers. Each of them pulls a rope to 
swing their bell. Then they wait for it to 


swing back before they can ring it again. 


BELLRINGING METHODS 


‘Change ringing’ - playing a sequence of 
‘changes’ (patterns) on a set of tuned bells 

in a precisely controlled way - originated in 
English churches. Church bells are massive, 
swinging objects with lots of inertia, so there 
are severe physical constraints to performance. 
Change ringing is circumscribed by conventions 
that make interesting music possible within 
these constraints - e.g., every bell sounds 
exactly once in each change, changes are not 
repeated, and each bell moves at most one 
position between consecutive changes. Despite 
these limitations, many thousands of different 
‘methods’ (performances) are possible - some 
involving a very large number of changes. 

An ‘extent’ or ‘full peal’ is a method in which 
every possible permutation of the the bells is 
heard exactly once. An extent on four bells has 
Ax3x2x] = 24 permutations; an extent on ten 
bells has over 3.6 million, and would take three 
months to play. 


FE, No-one could remember that. 


R They don’t have to — they just need 
to know the first change, then they 
use an algorithm to work out the next 
change. An algorithm, you know? 

It’s a set of rules or instructions, 
like computers use. And computers 
are everywhere — so algorithms are 
everywhere too. 


To make them humanly playable, methods are 
constructed in a logical manner out of a small 
set of simple rules. For example, in the method 
called ‘Plain Bob Hunting’ on four bells, the 
treble (bell #1) moves one position later in each 
change before reversing course: 

W234 3 2143 => 241 8 = 42 3 
while the other bells follow similar routes 
through the permutations. Each bellringer only 
needs to memorise a simple rule that produces 
successive changes.' 


MACHINE ALGORITHMS 


A bellringing method is an example of 

an algorithm, which can be defined as a 
computational procedure that produces 
unambiguous output from given input. More 
generally, an algorithm is a well-defined 
process for solving a specific problem. Powerful 
processors allow computers to run complex 
algorithms on large datasets at high speed.One 
type of algorithmic system consists of a (possibly 
very long) sequence of preprogrammed steps 
that reproduces human reasoning at speed and 
scale.2, More advanced algorithmic systems 
involve an aspect of ‘learning’, where machines 
‘discover’ the rules that best connect some input 
to an output by testing many different rules 

on large sets of training data. In supervised 
learning, the training data is structured (pre- 
interpreted) by humans, while in unsupervised 
learning, the system itself discovers categories 
and structure in the data.° 


WESTMINSTER ABBEY 


R Computers are everywhere — so 
algorithms are everywhere too. 


E What, even in those traffic lights? 


R The algorithm gets data from sensors in 
the road, and if it calculates that there’s 
going to be a traffic jam, it adjusts the 
timing of the traffic lights. 


E_ So the algorithm sees into the future...! 


R It guesses the future, more like, by 
seeing into the past. 


A rule that strongly links inputs to outputs 

can be regarded as ‘predicting’ the output. 
This ‘machine learning’ (ML) approach can 
handle broad problems and is extraordinarily 
powerful at tasks such as medical diagnosis 
and language translation. ML algorithms are 
embedded in Automated Decision Making 
(ADM) systems across sectors from energy and 
finance to traffic management, education, and 
criminal justice. 


BLIND JUSTICE 


In the criminal justice system, algorithms are 
used for tasks including risk assessment of 
individuals, predictive crime mapping, and 
verification of evidence. Automated risk profiling 
systems such as Durham Constabulary’s HART 
(Harm Assessment Risk Tool), or the commercial 
COMPAS (Correctional Offender Management 
Profiling for Alternative Sanction) software 
widely used in American courts,are marketed 

as more accurate, consistent, impartial and 
efficient than humans.’ System designers claim 
that by correctly identifying offenders at low risk 
of recidivism and keeping them out of prison, 
they prevent further criminalisation of these 
individuals and reduce the burden on the state. 
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THE SUPREME COURT 


R You know my aunt’s a judge in a court 
like that? She thinks that an algorithm 
could do her job! In the US, they use 
algorithms to decide if someone should 
get bail, or even how long a jail sentence 


should be. 


E Why would an algorithm be better than 
a person at that? 


R A judge might be prejudiced by the way 
a defendant speaks or looks, but an 
algorithm wouldn’t be. It would make 
decisions by looking into the past to see 
how similar people behaved. 


E But how is that fair? Just because 
someone’s similar to me doesn’t mean 
I'll do what they did. And anyway, 
how does an algorithm know that we’re 
similar?! 


However well-intentioned system designers 
might be, researchers have found grave 
problems with their implementation. Although 
neither HART nor COMPAS use the protected 
characteristic of ethnicity as a factor in assessing 
risk, they do use address data, which in 
segregated urban districts is an excellent proxy 
for ethnicity. HART has also been criticised 

for using data of unknown quality purchased 
from credit reference agencies. And a 2016 
investigation into COMPAS by ProPublica 

found that not only was the algorithm poor at 
predicting recidivism, but also that it consistently 
underestimated this rate for white defendants, 
and overestimated it for black ones.° 


R_ It’s not so easy to design fair algorithms. 
My aunt told me about a police 
algorithm that ended up making 
prejudices worse! 


E How? 


R It searched arrest records for patterns 
and used those used to predict where 
crimes might happen. But the police 
in the area were really racist. They’d 
cruise around black neighbourhoods 
arresting people for minor offences, 
and ignore similar crimes in white 
neighbourhoods. So the records made it 
look like black neighbourhoods were full 
of crime, and the algorithm would send 
more patrols there. 


E Oh, so they'd arrest even more black 
people for minor crimes...? 


R Exactly, and this algorithm — PredPol, 
i was calleds “Predictive Policing — 
would see these new arrests and send 
even more police there, making a bad 
situation even worse! 


PREDPOL: FEEDING BACK BIAS 


Some states are deploying anticipatory models 
of policing in the hope of transforming outcomes 
in the same way that preventitive medicine has 
revolutionised healthcare. Various ‘predictive 
policing’ systems, designed to identify where 
and when to send police patrols, have been 
deployed in the UK and US. Some early systems 
attempted to predict future crimes based 

ona theoretical (causal) model of why and 
where repeat offences occur, but most current 
systems, such as the American PredPol, use a 
machine learning approach to correlate crime 
occurrences with any other data available. 
Independent assessments of these systems 

have found major issues in the training data. 
Firstly, convictions or arrests are used as a 
proxy for actual crimes. Secondly, since training 
data is derived from historical records, past 
discrimination skews the output and can cause 
positive feedback loops - a neighbourhood 
that has historically been over-policed (because 
of racial prejudice, say) would show many 
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arrests per capita, and the system would direct 
more resources there, further entrenching 
discriminatory practices.° Moreover, because 
of their power and speed, ADM systems rapidly 
amplify even small biases. As in the case of 
sentencing algorithms, current implementations 
of predictive policing algorithms have systemic 
lacunze that must be addressed before the 
harder tests of social appropriateness and 
constitutional legitimacy can be applied. 


E But can’t we just check the algorithm to 
make sure it’s fair? 


R The company that made it keeps it 
secret, to stop others from copying it. 
It’s called a ‘black box’ algorithm — you 
input some data into it, and it outputs 
an answer, but you can’t look inside it to 
see how it got the answer. 


E That’s really shady. At least a person 
could tell you how they made a decision. 


The lack of transparency and accountability is a 
basic problem of algorithms such as COMPAS 
and PredPol. Being proprietary, details are not 
available for public scrutiny, and - in common 
with many ADM systems - there are no robust 
mechanisms for individual appeals. Predictive 
policing algorithms are so problematic that in 
2020, nearly 2,000 mathematicians addressed 
a letter to the American Mathematical Society 
urging their colleagues to boycott collaboration 
with the police. Even the U.S. Department of 
Justice has pointed to possible constitutional 
issues around due process with COMPAS.” 


DEPT. OF BUSINESS, ENERGY 
& INDUSTRIAL STRATEGY 


E Do you think that CCTV camera knows 
who we are? 


R Well, if it’s using a facial recognition 


algorithm, and is connected to a 
database with everyone’s faces, then yes! 
There’s no law against it here. Some 
places like San Francisco have banned 
facial recognition algorithms in public, 
because they’re just not accurate. 


E We should ban them here too then! 


R_ Too right! The worst thing is that 
HikVision has been building a mass 
surveillance system for the Chinese 
government to help them keep track of 
every citizen. It’s even supposed to be 
able to identify Uyghurs. 


E You mean the people that China’s 
imprisoning in camps? But then why is 
the UK buying Hikvision cameras — and 
putting them around Parliament? 


BIOMETRIC SURVEILLANCE 


There are at least 400 million surveillance 
cameras installed in China®, many of which 

are networked with a biometric database to 
allow the state to identify and track citizens by 
their faces. This country-wide facial recognition 
system provides data to the national social 
credit system. It also serves as a platform 

for running ‘minority analytics’ - in 2018, 

the state-controlled company Hikvision 
inadvertently revealed the existence of a 

tool for identifying ethnic groups such as the 
Uyghurs using facial recognition algorithms.’ 

In July 2019, the Parliamentary Foreign Affairs 
Committee reported that Hikvision is the primary 
technology provider at Uyghur internment 
camps. Concurrently, Hikvision was contracted 
to supply cameras to the Parliamentary Estate - 
an arrangement that not only lacks any probity, 
but also credibly threatens national security." 


Technology companies constantly oversell, 
however, as the measured performance of 
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facial recognition algorithms in the field 

shows. Between 2014 and 2019, several UK 
police forces conducted public trials of facial 
recognition cameras linked to the National 
Police Database and obtained poor results, 
with the rate of false positive identifications 
particularly high.'' But the issues of privacy, 
dignity and due process that arise with 
biometric surveillance are only compounded by 
such inaccuracies. 


DEPT. OF EDUCATION 
E What are you protesting about? 


A We couldn’t sit erams because of the 
pandemic, so our teachers graded us, 
and the Department of Education used 
an algorithm to check the grades. But 
it’s totally biased! 


B My school gave me all ‘A’s and then this 
stupid algorithm lowered them all to 
‘B’s, just because last year no-one in my 
school got all ‘A’s. And now I’ve lost my 
college place — how is that fair? 


C And for a kid in a top school, it didn’t 
matter if they worked or not — the 
teacher might give them a ‘D’, but if 
everyone scored higher than a ‘D’ the 
previous year, the algorithm would 
automatically boost the grade! 


E Those algorithm designers need to go 
back to school! 


BACK TO SCHOOL 


In August 2020, the Department of Education 
on Great Smith Street witnessed a remarkable 
public demonstration - perhaps the first 


anywhere to specifically protest an algorithm.'? 
The Covid-19 pandemic had disrupted 


school-leaving exams that were pivotal to the 
university admissions process. In lieu of exams, 
students were graded by an algorithm that 
took teachers’ evaluations, which tended to 

be inflationary, and moderated them using the 
particular school’s historical performance.'° 
However, the algorithm was poorly conceived. 
By forcing a grade distribution onto teachers’ 
rankings, many students found their grades 
significantly - and unfairly - deflated. A student 
evaluated with a high absolute grade, but 
ranked lowest in a class, might see their grade 
reduced to the absolute lowest grade achieved 
in the previous year. To make matters worse, 
teachers’ rankings were more strongly weighted 
for smaller cohorts, typically associated 

with independent (fee-paying) schools. 
Consequently, students at independent schools 
- already bastions of privilege - saw greater 
grade inflation.'4 As is typical, the appeals 
process was totally inadequate. Students took 
to the streets, and legal action was mounted 

by digital rights initiative Foxglove. Facing a 
judicial review, the government retracted the 
algorithm and grades were reverted to teachers’ 
evaluations.'° 


ADAM SMITH INSTITUTE 


Of course we don’t know what the Al 
algorithms do, that’s the point of using 
them.'° 


E Who’s that? 


R That was Tim Worstall from the Adam 
Smith Institute. It’s a think tank that 
advises politicians and businesses. 


E Wait — is he saying that it’s a good thing 
that I don’t understand how algorithms 
work? But if an algorithm’s going to 
change my grades or send me to jail, I’d 
want to know exactly why! 


AIS MYSTERIOUS WAYS 


Complex, successful algorithms and ADM 
systems are often described as exhibiting 
‘artificial intelligence’ (Al). Most current Al 
algorithms use a type of ML called ‘deep 
learning’, in which computation is carried out 
by a neural net. A neural net is an array of 
processing nodes whose interconnections are 
modelled on the way neurons connect in the 
brain; ‘deep’ refers to the large number of 
layers of nodes between input and output. 
Google DeepMind’s human-conquering chess 
and Go game engines and their protein-folding 
algorithm use this approach. Deep learning 
systems achieve striking successes without 
needing to be explicitly given any concepts or 
structure. On the other hand, the reasoning by 
which they produce particular outputs is usually 
opaque, even to their designers."” 


E But then whose fault is it if the 
algorithm makes a mistake? 


While not having explanations for a strategy 
or gameplay doesn’t undermine the entire 
culture of chess or Go, an inability to justify 
decisions that more viscerally impact human 
lives - through financial trading, or healthcare 
provision, or autonomous weapons systems 

- has a devastating effect on individuals and 
society. 


THE HOME OFFICE 


E That huge building with the coloured 
glass — looks like a fancy hotel. 


R You’re kidding — that’s the Home Office. 
They’re in charge of the police and 
prisons, they give out passports and 
visas. And guess what they’ve been using 
to decide who deserves a visa? 


E Don’t tell me — an algorithm. 


TEM iita( Crile en 
pakkGadge /f 


—— =GCHQ National Technical 


Assistance Centre 


Y ~~ ARE EIEDOY = 
fae ASS Ye \* we 
\ CONGR ~ 
\pe a Home Office 
pat 


j= .¢ ae 


€ 
\_ a Smith Institute 
Ly, 


Department of Education 


Lf Department of Business, Energy 
& Industrial Strategy 


eg ee 


\\ a Parliament Square 13 


A 


Westminster Abbey 


— 


R A secret one, naturally, which was 
actually designed to discriminate against 
people because of their nationality. It 
was totally racist — people from African 
countries found it really hard to get 
visas. But campaigners took the Home 
Office to court and proved that the 
algorithm was illegal, so they’ve had to 
stop using it for now. 


SECRET STREAMING 


In June 2019, the Financial Times revealed 

that the Home Office had been using a secret 
algorithm to process visa applications for 
several years.'® The Chief Inspector for Borders 
and Immigration had earlier criticised the 
algorithm for having the potential to become a 
de facto decision-making tool, and encouraging 
confirmation bias in visa caseworkers. The 
algorithm assigned a risk rating to applicants on 
the basis of nationality, and contained a vicious 
feedback loop. After increasing numbers of 
otherwise eligible African academics and artists 
were denied visas, the Joint Council on the 
Welfare of Immigrants worked with Foxglove to 
bring a legal challenge against the government. 
The Home Office withdrew the algorithm in 
2020," 


E So we can defend ourselves against 
unfair algorithms! 


R Sometimes! 


SUPERMARKET 


E Sainsbury’s Bank? I thought it was a 
supermarket! 


R It is a supermarket — and a bank. They 
sell pretty much everything you could 
want. Their website even kind of knows 
what you want... 


E They read your mind — with an 
algorithm? 


R You’d be amazed what they know about 
yOu... 


OMNICHANNEL RETAIL 


Multi-brand retailers such as Sainsbury’s pool 
data from across their operations to gain 
competitive advantage. The efficiencies and 
insights gained through patching together 
internal silos of data, for example from different 
points of the supply chain, are uncontroversial. 
What should be disturbing is how data 
harvested across different brands is pooled 

to build a multidimensional portrait of the 
customer - as someone who is no longer merely 
contracting for goods and services, but whose 
behaviour must be understood, predicted and 
nudged. The chief data and analytics office 

of Sainsbury's, Helen Hunter, wants ‘to stitch 
together what Nectar understands about how 
customers travel from the Nectar travel partners, 
how the bank can understand who's using 

travel money, how Argos can understand who’s 
buying luggage, and how Sainsbury's food can 
understand who's buying a load of sun cream.’”° 


Multiply this approach to business by the 
dozens of large retailers in the marketplace, and 
you, the customer, should perhaps be beginning 
to feel a little uneasy about just how interesting 
you appear. Not only must you navigate among 
multiple brands that may or may not be faces of 
the same corporation, but you are also courted 
on every platform and in every medium in a 
calculated manner. How does the retailer make 
their offer irresistible? - through an omnichannel 
strategy. Add something to an online shopping 
basket, and you might be sent coupons via 
social media. Forget to check out, and the next 
day you might be served ads on Youtube. A 
week later and still no purchase, and you might 
find physical marketing material in your mailbox 
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promoting an ancillary product. Return to the 
online store, and a deeper discount awaits. 
Upon purchase, the retailer expresses their 
gratitude by emailing an invitation to review the 
product for future discounts. If the email does 
not work, a last message over social media 
offers discounts for your friends. And after a 
respecttul silence, a new campaign starts via 
ads on the game you just downloaded. Do you 
understand your place in all this now? ?! 


MILLBANK HOUSE 


R There’s an office here called NTAC — 
National Technical Assistance Centre 
— which can listen to your phone 
conversations, read your emails, and see 
where you go on the Internet. All of that 
information is supposed to be private... 
it’s encrypted. But NTAC can crack that 
encryption — using algorithms. I heard 
that they’re even using a chess algorithm 
designed by Google DeepMind — not to 
play chess, but to intercept data on the 
Internet. 


CHECKMATE 


Of the millions of different algorithms that 
inhabit the software and processes of modern 
life - from mobile telephony and GPS 
navigation to keeping your utilities on and 
your fridge stocked - relatively few were 
crafted originally for their current applications. 
Abstraction is a feature of modern algorithms 
- and particularly of ML systems. It gives them 
their power, also their portability. A 2019 
Financial Times feature on GCHQ, the UK’s 
largest intelligence service, briefly mentioned 
a novel repurposing of Google DeepMind’s 
open-source chess-playing algorithm.* Instead 
of hunting for plays to win, this derived code 
hunts for the best points on the Internet at which 
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to eavesdrop on traffic. The office in which this 
technique was developed, NTAC in Millbank 
House, marks the end of this wander through the 
algorithms that infuse Westminster. 


E There are algorithms everywhere! 


R They’re part of our daily lives, so it’s 
important to know something about them, 
like what exactly they’re designed to 
and how well they do that, how fair and 
reliable they are, how their decisions can 
be challenged — and whether they should 
used for certain purposes at all.?3 


OMNICHANNEL STRATEGY 


Bank with us 
trade with us 
fly with us 
play with us 
Eat with us 
work with us 
buy with us 
sell with us 


The things that you want 
the things that you want 
the things that you need 


Keystroke logger 
IP logger 
GPS logger 
Searching 
searching 
surfing 
What’re you searching? 


Beach holidays 
Embarrassing problems 
Age-related intermittent 
back pain 
Hair loss? 
Anxious? 
Overweight? 
YouTube ads 
YouTube ads 
YouTube ads 


Have you considered — 
our wellness cure? 


The things that you want 
the things that you want 
the things that you need 


The voice of your choice 
to keep you warm at night 


Got a vet for your pet? 
We Insure your dog! 
Not protected by God? 


Ripe avocados on subscription 
Three square meals a day 
delivered to your door 


Workout? 
workout! 
workout! 

Personal best 

personal best 

personal best 

Your quantified self is what we want 

to predict your future health 


Early adopters 
early adopters 
early adopters 
Viral influencer! 


Like us 
tag us 
share us on Facebook 
On Facebook 
on Google 
on Instagram 
For an unmissable offer! 


And if you like the sound of that 
you re gonna love what’s coming next! 


LIS wO8 
Door to door 
Artificially Intelligent 
With a voice of your choice 


Our friendly digital assistant 
is the only service 
you will ever need 


Join our club! 
join our club! 
Join our club! 


(The free trial giveth 
and the terms and conditions 
taketh away) 
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